Yes, it serves a security purpose. It's a defence-in-depth measure.

The core reason it serves a purpose is that there are various bits of software that will validate some form of login credentials for a specified user and then use that user's login shell to run a command. Perhaps the most notable example is SSH. If you successfully authenticate as a user over SSH, SSH then launches the user's login shell (or uses it to run the command you provide, if you use the ssh [email protected] 'command to run' syntax).

Of course, ideally an attacker won't be able to authenticate as a user at all. But suppose the following situation happens:

Now your attacker can write an SSH public key to ~/.ssh/authorized_keys, then SSH in, and - boom! - they've got shell access to your server.

If the user instead has /usr/sbin/nologin as their login shell, then even after the attacker successfully writes the public key to authorized_keys, it's not useful to them. All it allows them to do is remotely run nologin, which isn't very useful. They can't get shell, and their attack has been mitigated.

In case this scenario seems very hypothetical, I'd like to note that I was targeted by precisely this attack at some point in 2015, around a year after I asked this question. Like a goddamned idiot, I'd left a Redis instance with no password open to the world. It got targeted by the Redis crackit attack, in which the attacker inserts an SSH public key into your Redis database, then sends a command telling Redis to write the contents of the database to .ssh/authorized_keys, then tries to SSH in. I was saved from the consequences of my own incompetence only by the fact that the maintainers of the redis-server Apt package (which I'd used to install Redis) had the wisdom to make it run Redis as a redis user who has no home directory or login shell; had it not been for them, my server would likely have been ransomwared or ended up part of some hacker's botnet.

That experience gave me some degree of humility and made me appreciate the importance of defence in depth, and, in particular, of not granting login shells to accounts that run webservers, databases, and other background services.